Example: Multilabel classification¶

---

This example shows how to use ATOM to solve a multilabel classification problem.

The data used is a synthetic dataset created using sklearn's make_multilabel_classification function.

Load the data¶

In [1]:

# Import packages
import pandas as pd
from atom import ATOMClassifier
from sklearn.datasets import make_multilabel_classification

# Import packages import pandas as pd from atom import ATOMClassifier from sklearn.datasets import make_multilabel_classification

In [2]:

# Create data
X, y = make_multilabel_classification(n_samples=300, n_classes=3, random_state=1)

# Create data X, y = make_multilabel_classification(n_samples=300, n_classes=3, random_state=1)

Run the pipeline¶

In [3]:

# Note that for multioutput tasks, you must specify the `y` keyword
atom = ATOMClassifier(X, y=y, verbose=2, random_state=1)

# Note that for multioutput tasks, you must specify the `y` keyword atom = ATOMClassifier(X, y=y, verbose=2, random_state=1)

<< ================== ATOM ================== >>

Configuration ==================== >>
Algorithm task: Multilabel classification.

Dataset stats ==================== >>
Shape: (300, 23)
Train set size: 240
Test set size: 60
-------------------------------------
Memory: 51.73 kB
Scaled: False
Outlier values: 29 (0.5%)

In [4]:

# Show the models that natively support multilabel tasks
atom.available_models()[["acronym", "model", "native_multilabel"]]

# Show the models that natively support multilabel tasks atom.available_models()[["acronym", "model", "native_multilabel"]]

Out[4]:

	acronym	model	native_multilabel
0	AdaB	AdaBoost	False
1	Bag	Bagging	False
2	BNB	BernoulliNB	False
3	CatB	CatBoost	False
4	CatNB	CategoricalNB	False
5	CNB	ComplementNB	False
6	Tree	DecisionTree	True
7	Dummy	Dummy	False
8	ETree	ExtraTree	True
9	ET	ExtraTrees	True
10	GNB	GaussianNB	False
11	GP	GaussianProcess	False
12	GBM	GradientBoostingMachine	False
13	hGBM	HistGradientBoosting	False
14	KNN	KNearestNeighbors	True
15	LGB	LightGBM	False
16	LDA	LinearDiscriminantAnalysis	False
17	lSVM	LinearSVM	False
18	LR	LogisticRegression	False
19	MLP	MultiLayerPerceptron	True
20	MNB	MultinomialNB	False
21	PA	PassiveAggressive	False
22	Perc	Perceptron	False
23	QDA	QuadraticDiscriminantAnalysis	False
24	RNN	RadiusNearestNeighbors	True
25	RF	RandomForest	True
26	Ridge	Ridge	True
27	SGD	StochasticGradientDescent	False
28	SVM	SupportVectorMachine	False
29	XGB	XGBoost	False

In [5]:

atom.run(models=["LDA", "RF"], metric="recall_weighted")

atom.run(models=["LDA", "RF"], metric="recall_weighted")

Training ========================= >>
Models: LDA, RF
Metric: recall_weighted


Results for LinearDiscriminantAnalysis:
Fit ---------------------------------------------
Train evaluation --> recall_weighted: 0.9124
Test evaluation --> recall_weighted: 0.8351
Time elapsed: 0.037s
-------------------------------------------------
Time: 0.037s


Results for RandomForest:
Fit ---------------------------------------------
Train evaluation --> recall_weighted: 1.0
Test evaluation --> recall_weighted: 0.8763
Time elapsed: 0.170s
-------------------------------------------------
Time: 0.170s


Final results ==================== >>
Total time: 0.269s
-------------------------------------
LinearDiscriminantAnalysis --> recall_weighted: 0.8351
RandomForest               --> recall_weighted: 0.8763 !

In [6]:

# Note that non-native multioutput models use a meta-estimator wrapper
print(f"Estimator for LDA is: {atom.lda.estimator}")
print(f"Estimator for RF is: {atom.rf.estimator}")

# Note that non-native multioutput models use a meta-estimator wrapper print(f"Estimator for LDA is: {atom.lda.estimator}") print(f"Estimator for RF is: {atom.rf.estimator}")

Estimator for LDA is: ClassifierChain(base_estimator=LinearDiscriminantAnalysis(), random_state=1)
Estimator for RF is: RandomForestClassifier(n_jobs=1, random_state=1)

Add custom multilabel models¶

To use your own meta-estimator with custom parameters, add it as a custom model. It's also possible to tune the hyperparameters of this custom meta-estimator.

In [7]:

from atom import ATOMModel
from sklearn.multioutput import ClassifierChain
from sklearn.linear_model import LogisticRegression
from optuna.distributions import CategoricalDistribution, IntDistribution

custom_model = ATOMModel(
    estimator=ClassifierChain(LogisticRegression(), cv=3),
    name="chain",
    needs_scaling=True,
    native_multilabel=True,
)

atom.run(
    models=custom_model,
    n_trials=5,
    ht_params={
        "distributions": {
            "order": CategoricalDistribution([[0, 1, 2], [2, 1, 0], [1, 2, 0]]),
            "base_estimator__max_iter": IntDistribution(100, 200, step=10),
            "base_estimator__solver": CategoricalDistribution(["lbfgs", "newton-cg"]),            
        }
    },
)

from atom import ATOMModel from sklearn.multioutput import ClassifierChain from sklearn.linear_model import LogisticRegression from optuna.distributions import CategoricalDistribution, IntDistribution custom_model = ATOMModel( estimator=ClassifierChain(LogisticRegression(), cv=3), name="chain", needs_scaling=True, native_multilabel=True, ) atom.run( models=custom_model, n_trials=5, ht_params={ "distributions": { "order": CategoricalDistribution([[0, 1, 2], [2, 1, 0], [1, 2, 0]]), "base_estimator__max_iter": IntDistribution(100, 200, step=10), "base_estimator__solver": CategoricalDistribution(["lbfgs", "newton-cg"]), } }, )

Training ========================= >>
Models: chain
Metric: recall_weighted


Running hyperparameter tuning for ClassifierChain...
| trial |     order | base_estimator__max_iter | base_estimator__solver | recall_weighted | best_recall_weighted | time_trial | time_ht |    state |
| ----- | --------- | ------------------------ | ---------------------- | --------------- | -------------------- | ---------- | ------- | -------- |
| 0     | [2, 1, 0] |                      130 |                  lbfgs |          0.8831 |               0.8831 |     2.813s |  2.813s | COMPLETE |
| 1     | [1, 2, 0] |                      150 |              newton-cg |          0.9091 |               0.9091 |     2.184s |  4.997s | COMPLETE |
| 2     | [2, 1, 0] |                      170 |              newton-cg |          0.8701 |               0.9091 |     0.085s |  5.082s | COMPLETE |
| 3     | [1, 2, 0] |                      200 |              newton-cg |          0.9221 |               0.9221 |     0.084s |  5.166s | COMPLETE |
| 4     | [2, 1, 0] |                      100 |              newton-cg |          0.8701 |               0.9221 |     0.078s |  5.244s | COMPLETE |
Hyperparameter tuning ---------------------------
Best trial --> 3
Best parameters:
 --> order: [1, 2, 0]
 --> base_estimator__max_iter: 200
 --> base_estimator__solver: newton-cg
Best evaluation --> recall_weighted: 0.9221
Time elapsed: 5.244s
Fit ---------------------------------------------
Train evaluation --> recall_weighted: 0.9021
Test evaluation --> recall_weighted: 0.866
Time elapsed: 0.101s
-------------------------------------------------
Time: 5.345s


Final results ==================== >>
Total time: 5.397s
-------------------------------------
ClassifierChain --> recall_weighted: 0.866

Analyze the results¶

In [8]:

thresholds = atom.rf.get_best_threshold()
print(f"Best threshold per target column: {thresholds}")

thresholds = atom.rf.get_best_threshold() print(f"Best threshold per target column: {thresholds}")

Best threshold per target column: [0.7, 0.69, 0.63]

In [9]:

atom.rf.evaluate(threshold=thresholds)

atom.rf.evaluate(threshold=thresholds)

Out[9]:

accuracy              0.5667
ap                    0.8893
f1_weighted           0.7274
jaccard_weighted      0.6271
precision_weighted    0.8269
recall_weighted       0.6495
auc                   0.9213
Name: RF, dtype: float64

In [10]:

# Use the target parameter in plots to specify which target column to use
atom.plot_roc(target=2)

# Use the target parameter in plots to specify which target column to use atom.plot_roc(target=2)

In [11]:

# When the target parameter also specifies the class, use format (column, class)
atom.plot_probabilities(models="chain", target=(2, 1))

# When the target parameter also specifies the class, use format (column, class) atom.plot_probabilities(models="chain", target=(2, 1))

In [12]:

with atom.canvas(figsize=(900, 600)):
    atom.plot_calibration(target=0)
    atom.plot_calibration(target=1)

with atom.canvas(figsize=(900, 600)): atom.plot_calibration(target=0) atom.plot_calibration(target=1)